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IntroJuction 

Ihis paper describes activities concJucted as part of a project sponsored 
under contract with the Office of Educational Research and Utprovenient (OERI) . 
Hie initial request for proposals issued by OERI (#R-86-00l7) contained the 
following general description of the goals of the work. 

Each stucfy will focus on a single discipline or field and 
determine how indicator (s) can be constructed so as to 
account for suniraative urdergraduate learning (of content 
methods, assunptions, etc.) in that field, and c- help a^er 
the question of v^t it means to be educated in a discipline. 

Ihere were 19 responses to the REP representing a divei - cross-section of 
disciplines. Five proposals were funded. Besides our project on physics the 
other four studies represented the disciplines of biology, chemistry, conputer 
science, and mechanical engineering. 

The RFP listed a series of specific tasks to be acconplished by each 
contractor, including the review arxJ analysis of existing instruments and 
methods for assessing "summative learning" in the discipline. Ihis paper will 
focus on an analysis of ■:ne Graduate Record Examination in Physics as a specific 
existing indicator of learning outcomes in that undergraduate major. 

Ihe Undergraduate Curriculum in Physics 

The definitive source of informtion concerning the undergraduate physics 
curriculum in U.S. colleges is provided by the Committee on Professional Concerns 
and Undergraduate Education of the American Association of Physics Teachers 
(MET) . Ihe committee has prepared a booklet, AAPT Guidelines for the ReviP>w nf 
Baccalaureate Physics Prnq rams v^ch is available through MPI. Ihe most recent 
version was published in September, 1987. 

Ihe section entitled "Curriculum" suggests that the undergraduate curriculum 
begin with 

"an elementary course that has at least five subsections: 
Mechanics, Waves, Heat and Ihermoc^Tiamics, Electricity and 
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Magnetism, and Optics. A time commitment of at least 

two semesters is required to teach the five standard 
subsections; a time frame of three semesters is a LeLuer 
choice if Modem Etiysics is to be included." 

Witli respect to more advanced xandergraduate courses that a physics major 

ought to take, the Guidelines state, "There should be a vigorous, advanced 

treatment of tcincs in Mechanics, Electricity and Magnetism, Thermodynaiaics and 

Statistical Mechanics, qptics. Quantum Hiysics, and Experimental Physics." More 

specifically, the Guidelines present a detailed listing of topics suggested for 

inclusion in each of these areas. Figure 1 presents the detailed listing of 

topics recommended for inclusion in each of these six advanced courses. 



Insert Ficaire 1 about here 



The Graduate Record Examination Etiysics Test 

The Hiysics Test is described as follows in Practicing to Take the GRE 

Physics Test published by Educational Testing Service in 1986. 

The test consists of about 100 five choice questions, some of 
which are groi:ped in sets and based on sucii materials as 
diagrams, graphs, e>^)erijnental data, and descriptions of 
physical situations. 

Ihe errphasis of the test is on the students' firm grasp of 
fundamental principles and their ability to apply and 
understanding of them in the solution of problems. Most test 
questions can be answered on the basis of a mastery of the 
first three years of undergraduate physics. 

Ihe approximate percentages on content topics have been set 
by the committee of examiners to reflect its judgment about 
the relative enphases placed on these topics in most 
undergraduate curricula. These percentages are as follows: 
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. Percentage 
?°P^^, . of Questions 

1. Classical mechanics 18 

2. Func3ainentals of electromagnetism, 

including Maxwell's equations 18 

3. Atomic fSiysics 15 

4. Physical optics and wave phenomena in 

5. Quantum mechanics U 

6. Special relativity 7 

7. Thermodynamics and statistical mechanics 7 

8. laboratory methods 5 

9. Advanced topics (lagrangian and Hamiltonian 
mechanics, nuclear and particle physics, 

solid state physics, and miscellaneous) lo 

The time limit for the test is 170 minutes so examinees have less than two 
minutes per item. Based upon the responses of 10,625' examinees who took the test 
(Form GR8677) beta-een October 1, 1981 and September 30, 1984, the median raw 
score (corrected for guessing) was 37. The average item difficulty for this 
grocp of examinees was .45 with individual item difficulties ranging from .14 to 
.90. 

Procedure 

Textbook Survey 

The approach taken in this stucfy eirphasizes the analysis of the 
undergraduate curricula at a sanple of colleges to determine the degree of 
consensus across institutions with respect to general program requirements and 
specific course offeringr As previously noted, the AAPT guidelines suggest a 
fairly "standard" curricoxum. In particular, we wished to identify and analyze 
commonly employed physics textbooks since the content of the undergraduate 
curriculum is determined to a large extent by the textbooks in common use. 

Textbooks were ssi.ected as a measure of the curriculum taught because they 
provide a basis for inter-institutional corrparison vAiich no listing of course 
titles or descriptions can. Further, we suspected (and our survey bears out) 
that a relatively small number of textbooks represents the curriculum taught in 
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undergraduate programs throu^out the cxDuntry fairly wcill. We assume in this 
procedure that the textbooks assigned describe what is actually taught. We have 
no systematic check on this assumption, thou^ it is consistent v/ith qualitative 
inpressions and anecdotal information. 

To (±>tain a t>dii{)le of schools for the survey, we selected 80 schools 
representative of four categories of schools in the U.S.: 20 highly ranked 
schools with graduate programs (TG) , 20 hi^y ranked schools without graduate 
programs (TOG) , 20 other randomly selected schools with graduate programs (RG) , 
and 20 other randondy selected schools without graduate programs (RUG) . 

To choose the highly ranked (TG) schools offering graduate programs, we took 
the tcp 20 as listed in the 1982 ranking sponsored by the Conference Board of 
Associate Research Cduncils (Jones, et.al., 1982). To choose the 20 "other" (PG) 
schools offering graduate programs, we made a random selection of 20 schools from 
the remaining schools offering graduate programs in the American Institute of 
Physics (AIP) listing of institutions vMch offer undergraduate degrees in 
physics (Ellis, 1986) . Each school was wei^ted by the number of physics major 
qraduates in 1985 in makiiig the selection. 

To select the 20 hi^y ranked schools (TOG) not offering physics graduate 
programs, we used the list of 50 liberal arts colleges participating in the 
Second National Conference on "The Future of Science at liberal Arts Colleges" 
(Carrier, et.al., is '7) and made a random selection of 20 schools from it, 
wei^ting each school by the number of physics mjor graduates in 1985. (This 
procedure was forced on us by the fact that no reliab.,e ranking of undergraduate 
physics programs appears to be available.) Finally, we made a random selection 
of 20 more schools not offering graduate programs (HX5) from the AIP list, 
wei^ting the schools in the same way. 
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Each of the 80 schools was sent a letter esqjlaining the project and a form 
asking for 1) the number of jiiysics majors graduated in the preceding year, 2) a 
3ist of courses taken by physics majors and the number of physics majors taking 
the course in 1985-86, and 3) the textbook used in each course listed. Each 
school which did not respond was telephoned at least once. 
Indices of 6RE an d Curriculum ConaruennP- 

In the next phase of the study ve used the textbook survey data to determine 
quantitative measures of the extent to which the GPE Fhysics Test measures 
outcomes represented by textbooks assigned to physics majors. To ej^lain the 
measures we imagine the material tested by the GPE to be one set labelled GPE and 
the material taught from the textbooks of the curriculum as another set labelled 
CURR as shown in Figure 2. We designed an index to estimate the quantity 

Rl = GPE n CURE^GPE 
and another index to estimate the quantity 
R2 = GPE n OM^CURR. 
Since these two indices represent part/whole relationships each takes on 
values between zero and one. Rl may be thou^t of as the extent to which the 
curriculum teaches students x*iat they need to know to perform well on the GPE. 
R2 may be said to be the extent to which the GRE tests the material which is 
taught from the texts of the curriculum. Put another way, Rl measures the extent 
to which the curriculum "covers" the GRE wliile R2 is the extent to which the GRE 
"cavers" the curriculum. 



Insert Ficaire 2 about here 



To obtain an operational estimate of the correspondence of textbook and GRE 
content, two sets of expert ratings were collected. First, two physics 
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professors with teaching esqjerience in each of four subfields (Mechcmics, 
Electricity and Magnetism, Ihennodynaitiics/Statistical Mechanics, and Modem 
Physics/Quantum Mechanics) were provided with copies of GRE items from Form 
GR8677 and with textbooks our survey revealed to be in most common use. Each of 
the ei^t professors was asked to answer the question, "To vihat extent would GRE 
question q be appropriate (or too easy) for the final examination in a course 
tau^t from textbook t?". Ratings for individual items in each subfield were 
made on a scale from zero to one in inter/als of .1. These ratings were enployed 
in the calculation of the index. 

A second group of eight ejqperts (again, two physics professors in each of 
the four designated subfields) was used to c±)tain ratings of the degree to vAiich 
the textbook content "cavers" the GRE. A list of 20 end-of-<iiapter questions was 
selected at random from each of the textbooks. Ttie questions and a copy of the 
GRE (Form GR8677) were provided for eacii professor. Ohey were asked to answer 
the question, "If you taught a course to prepare students for the GRE, how 
appropriate woizld this question be on a final examination for that course?". The 
rating scale was the same as previously described for the first group of experts. 
These ratings were used in calculating the R2 index. 

Specifically, the two desired indices were defined as: 

1) Ri = 1/N 2S Fi(q,t) f(t) 

qt 

and 

2) R2 = 1/20 Z2 F2(q,t) f(t) 

qt 

where: N = number of GRE items in a given subfield. 

Fi(q/t) = average rating of two experts in the first panel of raters. 
F2(q/t) = average rating of two e:qperts in the second panel of raters. 

f (t) = the proportion of the students in our survey sample using textbook t. 
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A sijTplif ied illustration of the calculation of the % index is presented in 
Table 1. In this hypothetical situation, there are ten GRE iteins rated by each 
of the two ej5>erts with relation to three textbooks. The steps in calculating 
for this set of data are presented in detail in the figure. 

Insert Table 1 about here 

' Results 

Textbook Survey 

Usable data were obtained from 59 schools, or 74% of the saitple. The 
number of schools responding in each category is shown in Table 2. 

With regard to textbook adoptions, we found that a small nxMber of texts in 
each subfield accounts for the great itajority of the students at institutions in 
our sample. To keep the number of texts for further analysis manageable, we 
decided to restrict attention to texts used by ten percent or more of the 
students who were physics majors in the schools responding to our survey. Using 
this criterion, the number of texts retained and the total percent of students 
using these texts are as presented in Table 3. 

Insert Tables 2 and 3 about here 

The total percent exceeds 100 in the Modern/Quantum Ehysics due to the fact 
that students frequently take iDore than one course in that catfgory. 
Reliability of Expert Ratings 

The correlation between the ratings of the two experts in each subfield for 
each type of rating was calculated for each textbook. The range of correlations 
and the median correlation for each subfield and of rating are given in 
Table 4. 
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Ihe results in Table 4 reveal that the reliability of ratings of the degree 
to vAiich text±)ooks "cxjver" the GRE items is quite respectable although there is 
cxDnsiderable variation both among subfields and aiixDng individual textbooks within 
a particular subfield. 

The reliabilities for the ratings of the degree to which the GRE "covers" 
the textbook items are substantially lower. Inspection of these ratings shows 
that, on average, the ratings are much lower than are the first set of ratings. 
Indeed, the average rating for the 20 items randomly selected frcsn textbooks was 
less than .20 (on a scale from zero to one) for over 40 percent o£ the ratings. 
At least half of the textbook items were rated zero over one third of the time. 
Distributions with such extreme skews and small variabilities obviously place 
constraints i:pon correlations among ratings. 

A plausible ejqplanation for such lew ratings of the textbook items is that, 
being randomly selected, such items frequently deal with concepts, methods, and 
problems which are not of central importance v*ien judged in the context of the 
larger subfield. Obviously, such items are not of the type that are likely to be 
selected for inclusion on the GRE! 

Insert Table 4 about here 

Indices of Content Validity of the GRE 

Rl indices were calculated separately by subfield and by type of institution 
in our survey sample. The results are shown in Table 5A. The R^ values indicate 
that the curriculum in Mechanics generally prepares students well for the GRE, 
regardless of institution type. However, the courses in Electricity and 
Magnetism seem to be covering only about 60 percent of the relevant material on 
the GRE. A more detailed analysis reveals that there are a significant number of 
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questions on optics in the GRE while the typical curriculum does not give great 
emphasis to thir topic. 

For the ^wo remining subfields, the values for institutions with 
gracJuate level programs are quite high but the values for ijistitutions with 
stricUy undergraduate programs are much lower. A close inspection of our data 
suggests that this arises because schools without graduate programs offer or 
require fewer courses in these two subfields. Also, in the absence of a graduate 
level curriculum and - ^«arch program, fewer undergraduate students are lik^y to 
take courses in Statistical Mechanics/Ihermodynamics and Modern/Quantum physics 
even when they are available. 



Insert Tables 5A and 5B about here 



Ihe R2 indices were also calculated separately by subf ield and institution 
type. These are presented in Table 5B. Ihe most striking feature of these 
results is the much Icwer magnitude of the values in relation to the values in 
Table 5A. About 40 percent of the textbook content in Mechanics and Electricity 
and Magnetism is represented by the GRE. Ihe figure is less than half of that 
for the other two fields. Also, there is a uniformity of results across 
institutions with each subfield. Hwever, these results need to be interpreted 
with caution in view of the lew reliabilities previously noted. 

Conclusions 

Ratings of GRE items indicate that the commonly used texts in Mechanics do a 
good job in preparing students for GRE items in that subfield, but texts in 
Electricity and Magnetism do less well in that regard. In the other two 
subfields there is a clear difference between institutions with graduate programs 
and those without graduate programs. Ihe difference is especially striking in 
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Modern/Quantum Hiysics, a subfield vMch represents over one third of the GRE 
items, ^niis suggests that students attend strictly undergraduate 
institutions may be at a substantial disadvantage on the GRE Ehysics Test. 

The ratings of items randomly selected frcM the most commonly adopted texts 
suggest that sucdi items are not well represented by items on the GRE. The 
ratings were so losr/ that tlie inter-rater reliabilities were generally Iot and 
quite variable from one text to another. It is not possible to state whether 
this results from the random selection process or from a "true" misnatch bet^veen 
the texts and the GRE. To clarify this issue, it would be necessary to use 
expert judgments in selecting textbook items that are "representative'' of 
outcomes considered to be most inportant. Ihis is presumably hcM the GRE items 
were selected. 
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Figure 1 

Description of Recommended Content; for Advanced 
Undergraduate Courses for Physics Majors^ 



Mechan/csiJhe mathematical level of ihe course should require the use of differen- 
tial equations. Central forces should be studied through at least the development 
of Kepler's laws. The study of systems of particles should pursue the conse- 
quences of the conservation of energy, momentum, and angular momentum; the 
latter should include the use of the ineriia tensor. The analysis of rigid body motion 
should include the application of Euler s equations. Lagrangian mechanics should 
be treated in sufficient depth for its application to small oscillations and coupled 
oscillators. 

Eiectricity and Magnetism: The mathematical level of this course should require 
the use of field operators and vector integral theorems. The treatment of elec- 
trostatics should encompass Coulomb's law; the electrostatic lield and potential; 
the Laplace and Poisson equations; electric dipoles; multipole expansions of poten- 
tials; electrostatic energy and force; capacitance; polarization; dielectr.cs; and the 
electric displacement field. The topics of electric current. Ohm's law, and the conti- 
nuity equation will lead to discussions of magnetism, including the magnetic induc- 
tion field; the Biot-Savart law; Ampere's law; magnetic energy, force, and torque; 
magnetization; and the magnetic field. Maxwell's equations should be considered 
essential components of this course and they should be applied to simple geome- 
tries (e.g., plane waves ii« infinite, non-conducting medium). Complex waves 
could also be introduced. li i permits, rclativistic electrodynamics could be 
briefly considered. 

Thermodynamics/Statisucal Mechanics. A thorough grounding in the concepts of 
temperature, work, specific heat, compressibility, and entropy should result from 
this course. The laws of thermodynamics, from the zeroth to the third should be 
discussed, with a thorough discussion of the import of the second law. The four 
thermodynamic generating functions (internal energy, enthalpy, Helmholtz function, 
and Gibbs function), together with Maxwell s relations, should be used to solve 
practical probiecns such as gas laws, engines, radiation, and phase transitions. The 
kmetiC theory of gases, partition functions, and, as possible, ensembles are cov- 
ered. Here is the student s first exposure to Maxwell-Boltzmonn. Fermi-Dirac, and 
Bose'Einstein statistics 



Optics Both geometrical and physical optics should be included. Enough time 
should be spent on thin lenses and mirrors to Drovde an understanding of simple 
optical systems and such concepts as magnification, entrance and exit pupils, and 
stops. The treatment of physical optics should include a discussion of two-beam 
and multiple-beam interference, diffraction at apertures, and the application of 
those principles to simple interferometers, double-slit diffraction, the diffraction 
grating, and diffraction-limit resolution Polarization and reflection should also be 
included. As time permits, there should be selective coverage of thick lenser. lens 
aberration, lens design, vision, color, ray tracing, birefrigence, spectroscopy, scat- 
tering, transfer functions, radiometry, and photometry. Some mention of lasers, 
holography, fiber optics, gradient-index optics, phase conjugation, and optical 
computing would tie the course to current technological development 

Quantum Physics. The historical founoations of quantum physics, blackbody radia- 
tion. Compton scattering, the Davisson-Germer experiment, and the Bohr- 
Sommerfeld model of the atom should be established (if not previously given the 
student in a modern physics course). The quantum physics course should include 
in-depth applications of the Schrodinger equation to one-dimensional problems 
such as the square-well potential, barrier scattering and tunneling, and the harmon- 
ic oscillator. The treatment of .quantized angular momentum should include some 
elementary work with operator methods and commutators. Three-dimensional 
problems should, at a minimum, describe the hydrogen atom and should include 
relativistic corrections. If this course is extended into s second semester there 
should be applications of essential quantum concepts to major fields of contempo- 
rary physics, e.g.. multiple particle wavefunctions vis-a-vis elementary t^uark 
models, shell theory applied to nuclear models, group theory, and matrix methods 
applicable to the theory of solids. 

Experimental Physics . The goal of thi& laboratory course »s to give the student ex- 
perience with real-world apparatus such 35 lasers, high field magnets, detectors, 
radioactive sources, vacuum equipment, and sophisticated electronics (at the level 
of lock-in amplifiers and multichannel scalers). The schedule should be a blend of 
classic experiments illustrating concepts from electricity and magnetism and quan- 
tum physics (the Fianck-Hertz experiment. Zeeman effect with ions, measurement 
of the speed of light, etc.) as well as experiments designed to convey the flavor of 
contemporary experimental physics. Examples of the latter are experiments on 
tunnel junctions, angular correlation of gamma rays, nuclear decay spectroscopy, 
and magnetic resonance spectroscopy. Special attention should be given to written 
communication of scientific information (see ihe AfP Style Manual), 



^ 0 ' en from the 1 987 AAPT Guidelines for the Review of Baccalaureate Physics Programs . 
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Figure 2 

Representation of the Relationship Between 
the Content of the 6RE Physics Test 
and the Content of the Undergraduate 
Curriculum in Physics 
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Table 1 



Illustration of the Coroputaticxi of 
Rl for a Hypothetical Setting 
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f(ti) = .30 f(t2) = .40 f(t3) = ,20 



% = 3^ (.5715) = .5715 (From Ecyi 1, p. 6) 

Alternatively, 

Rl = .30 (.725) + .40 (.550) + .20 (.670) = .5715 
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Table 2 
Summary of Survey Responses 

Category of School # of Schools # Phvs Grads, '86 
TG 14 

RG 13 222 

TOG 19 208 

RUG 13 78 



Table 3 

Summary of Textiook Usage 

# Texts Used itotal % Students 

Curriculum Category By 10% or More UsirKrr These Texts 

Mechanics 4 92 

Electricity/Magnetism 5 75 

Ihermodynamics/ 2 80 

Statistical Mechanics 

ModenVQuantum 5 117 



Table 4 

Summary of Correlations for Ebqpert Ratings 
Subfiald 



Type of 
Rating 

GRE Items 

Text±>ook 
Items 



Mechanics 



Range Mdn 
' .51 to .86 .66 
"•13 to .58 .32 



Electricity/ 

Magnetism 

Range Mdn 

.63 to 1.00 .80 

-.23 to .44 .34 



Ihermoc^niamics/ 

Statistical Mech Modem/Quantum 
Range Mdn 

.46 to .87 .71 

.31 to .64 .42 



Range Mdn 
.84 to .87 .80 
.07 to .45 ^26 
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Table 5A 

Rl Values by Stibfield and Institution Type 



Subfield 


TG 


Type of Institution 
TUG RG 


RUG 


Mechanics 


0.79 


0.76 


0.84 


0.84 


Electricity and Magnetism 


0.63 


0.58 


0.59 


0.61 


Statistical Mechanics 
and Ihermodynamics 


0.77 


0.61 


0.82 


0.63 


Modern/Quantum 


0.82 


0.50 


0.85 


0.47 



Table 5B 

R2 Values by Stibfield and Institution Type 



Subfield 


TG 


Type of Institution 
TUG RG 


RUG 


Mechanics 


0.44 


0.38 


0.48 


0.45 


Electricity and Magnetism 


0.44 


0.42 


0.42 


0.40 


Statistical Mechanics 
and Ihermoc^mamics 


0.17 


0.15 


0.16 


0.18 


Modeny Quantum 


0.24 


0.14 


0.25 


0.14 
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